Qualcomm: decide graph-output publishing per output, not per node - #22011
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22011
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit a00f3e5 with merge base baafd7e ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
psiddh
added
the
module: qnn
This PR needs a
|
is_graph_output() took only a node, so for a multi-output op it answered one question -- "does anything from this op leave the partition?" -- and applied that single answer to every output. One escaping output published all of them as QNN_TENSOR_TYPE_APP_READ, including outputs with no consumer at all. The ExecuTorch partitioner meanwhile wires up only the getitems that have live users, so the two halves of the same export disagreed about how many outputs the graph has, and nothing cross-checked them. The QNN graph then declares more outputs than ExecuTorch binds, and the two are paired by position, so the extra entries shift everything after them. Take an optional output index and, for a multi-output node, follow only that index's getitem. Callers passing no index keep the previous behaviour, so the remaining call sites are unaffected. define_tensor already has wrapper_idx and already passes it to get_tensor_name, so thread it into get_tensor_type as well. op_custom_op already asks for QNN_TENSOR_TYPE_NATIVE on every output; get_tensor_type was overriding it. Also make is_graph_output tolerate users whose target has no __name__ -- call_module targets are plain strings, and the previous form raised AttributeError on them. Verified on x86 with a six-output custom op at a partition boundary with indices 0, 1 and 4 consumed: APP_READ goes from 6 to 3, matching the consumed set exactly. On a model returning those outputs out of order the unfixed version segfaults, so this is a crash fix rather than only a correctness one. Diagnosed by Min Guo, who is carrying an equivalent local patch; this makes it a first-class fix so any multi-output op benefits. Authored with assistance from Claude (Claude Code).
076d84b to
6463f20
Compare
qti-horodnic
left a comment
qti-horodnic
left a comment
There was a problem hiding this comment.
Thanks for the change, LGTM. Since this changes output publication for every multi-output builder using wrapper_idx, do you mind running one of the on-device tests in test_qnn_delegate (e.g. topK) to make sure there are no regressions?
Left a couple of other comments, one is minor, feel free to address the other one in a separate PR if you want to keep this one contained.
| any( | ||
| user.op == "output" | ||
| or user.target.__name__ == "getitem" | ||
| and is_graph_output(user) | ||
| for user in tensor.users.keys() |
There was a problem hiding this comment.
Might be a bit of a scope creep, but since this method has the same issue as the one fixed above, can we apply the same getattr(user.target, "__name__", "") change here while you're modifying the file?
There was a problem hiding this comment.
Good call. same getattr(user.target, "name", "") guard applied to is_mutable_buffer_output.
| node, self.edge_program | ||
| ) | ||
| is_output = is_graph_output(node) | ||
| is_output = is_graph_output(node, wrapper_idx) |
There was a problem hiding this comment.
Minor: wrapper_idx is not necessarily always an output index (e.g. in op_scatter_elements it's just a scratch variable). Should we add a check to pass it here only if an output index? Or we can add a comment stating the assumption
There was a problem hiding this comment.
Yeah.. op_scatter_elements is exactly that case... Made it structural instead of a comment
…n tensors Review follow-ups on the per-output graph-output change. wrapper_idx is not always an output index. op_scatter_elements uses it to key a scratch tensor built from a different node, where it carries no output meaning. Pass it to get_tensor_type only when the tensor being defined belongs to the node itself, which is what every genuine multi-output builder does (op_topk, op_sort, op_unbind, op_split_with_sizes, op_custom_op all call define_tensor(node, node, ..., wrapper_idx=i)). Also apply the same getattr guard to is_mutable_buffer_output, which had the identical AttributeError on users whose target has no __name__.
While I test the output on the real device, let me know if this helps.. I ran the topK builder path on x86: topk raw index out consumed=[0,1] idx0=APP_READ idx1=APP_READ Last row is the behaviour change , previously both were APP_READ. topK, sort, unbind, split_with_sizes and scatter_elements all lower cleanly. |
is_graph_output() took only a node, so for a multi-output op it answered one question -- "does anything from this op leave the partition?" -- and applied that single answer to every output. One escaping output published all of them as QNN_TENSOR_TYPE_APP_READ, including outputs with no consumer at all.
The ExecuTorch partitioner meanwhile wires up only the getitems that have live users, so the two halves of the same export disagreed about how many outputs the graph has, and nothing cross-checked them. The QNN graph then declares more outputs than ExecuTorch binds, and the two are paired by position, so the extra entries shift everything after them.
Take an optional output index and, for a multi-output node, follow only that index's getitem. Callers passing no index keep the previous behaviour, so the remaining call sites are unaffected. define_tensor already has wrapper_idx and already passes it to get_tensor_name, so thread it into get_tensor_type as well.
op_custom_op already asks for QNN_TENSOR_TYPE_NATIVE on every output; get_tensor_type was overriding it.
Also make is_graph_output tolerate users whose target has no name -- call_module targets are plain strings, and the previous form raised AttributeError on them.
Verified on x86 with a six-output custom op at a partition boundary with indices 0, 1 and 4 consumed: APP_READ goes from 6 to 3, matching the consumed set exactly. On a model returning those outputs out of order the unfixed version segfaults, so this is a crash fix rather than only a correctness one.
cc @cbilgin